Flat File Processing

Processing a flat file is most frequently done over SFTP, often against a server in between two parties. Pathify habitually processes provisioning files from its customers through this mechanism, for instance.

A non re-entrant timer is set for how frequently the file system should be checked for changes.

If new files exist, they will be processed and then moved aside so that they are not detected again.

Files are tested against a mask - a pattern they must satisfy. For instance, "starting with 'users'" and having the extension "xml".

Each matching file content is loaded into a queue for processing.

Another behavior observes this queue, and is triggered once it becomes non-empty.

This behavior consumes the queued data, transforms it to a standard payload so that standard downstream processes can be applied, and acknowledges completion, deleting it from the queue.